Automatic, or computer, scoring of student essay examinations has been studied extensively by Professor Ellis B. Page and others in connection with Project Essay Grade (PEG). See, e.g., Page, E.B. (1994), Computer Grading of Student Prose, Using Modern Concepts and Software, Journal of Experimental Education, 62(2), 127-142. Although the details of the PEG systems and methods have apparently not been published, the available information indicates that the PEG system scores essays on the basis of approximations, or "proxes", of intrinsic criteria such as diction, grammar, and punctuation. For example, diction may be measured by variance in word length, and fluency by the number of words. The measured proxes are put into a scoring equation and a regression operation is performed to maximize the correlation of the computer score with that of human judges. Thus, e.g., word count may be transformed into its fourth root because, although human judges want students to produce a respectable amount of writing, writing beyond the point deemed "enough" is regarded less favorably. Since the fourth root of essay length flattens rapidly as essay length increases, it provides a useful predictor of the scoring of a human judge.
Such automatic essay scoring techniques are known to have problems with coachability and defensibility. Coachability refers to the relative ease with which a student can be coached to "trick" the system into giving her a better score than she deserves. Defensibility refers to the tendency of the system to unfairly downgrade an essay that has the correct content but lacks the features sought by the scoring system. These problems relate to the system's failure adequately to consider the content of an essay. The "content" of an essay, as that term is used herein, refers to the relevant concepts contained in the essay. The relevant concepts are predefined in view of the domain of knowledge with which the essay is concerned, and are used to define prescribed rules for scoring the essay. The particular essay scoring rules are not a part of the present invention, and are known to those skilled in the art of standardized testing. Further background information on computerized essay scoring is provided in the papers submitted with the Information Disclosure Statement filed herewith.